Deep Neural Network Inference Research Articles

SMaLL: Software for Rapidly Instantiating Machine Learning Libraries

Interest in deploying deep neural network (DNN) inference on edge devices has resulted in an explosion of the number and types of hardware platforms that machine learning (ML) libraries must support. High-level programming interfaces, such as TensorFlow, can be readily ported across different devices; however, maintaining performance when porting the low-level implementation is more nuanced. High-performance inference implementations require an effective mapping of the high-level interface to the target hardware platform. Commonly, this mapping may use optimizing compilers to generate code at compile time or high-performance vendor libraries that have been specialized to the target platform. Both approaches rely on expert knowledge across levels to produce an efficient mapping. This makes supporting new architectures difficult and time-consuming. In this work, we present a DNN library framework, SMaLL, that is easily extensible to new architectures. The framework uses a unified loop structure and shared, cache-friendly data format across all intermediate layers, eliminating the time and memory overheads incurred by data transformation between layers. Each layer is implemented by specifying its dimensions and a kernel , the key computing operation of that layer. The unified loop structure and kernel abstraction allows the reuse of code across layers and computing platforms. New architectures only require a few hundred lines in the kernel to be redesigned. To show the benefits of our approach, we have developed software that supports a range of layer types and computing platforms; this software is easily extensible for rapidly instantiating high-performance DNN libraries. An evaluation of the portability of our framework is shown by instantiating end-to-end networks from the MLPerf:tiny benchmark suite on five ARM platforms and one x86 platform (an AMD Zen 2). We also show that the end-to-end performance is comparable to or better than ML frameworks such as TensorFlow, TVM, and LibTorch.

ECHO: Energy-Efficient Computation Harnessing Online Arithmetic—An MSDF-Based Accelerator for DNN Inference

Deep neural network (DNN) inference demands substantial computing power, resulting in significant energy consumption. A large number of negative output activations in convolution layers are rendered zero due to the invocation of the ReLU activation function. This results in a substantial number of unnecessary computations that consume significant amounts of energy. This paper presents ECHO, an accelerator for DNN inference designed for computation pruning, utilizing an unconventional arithmetic paradigm known as online/most significant digit first (MSDF) arithmetic, which performs computations in a digit-serial manner. The MSDF digit-serial computation of online arithmetic enables overlapped computation of successive operations, leading to substantial performance improvements. The online arithmetic, coupled with a negative output detection scheme, facilitates early and precise recognition of negative outputs. This, in turn, allows for the timely termination of unnecessary computations, resulting in a reduction in energy consumption. The implemented design has been realized on the Xilinx Virtex-7 VU3P FPGA and subjected to a comprehensive evaluation through a rigorous comparative analysis involving widely used performance metrics. The experimental results demonstrate promising power and performance improvements compared to contemporary methods. In particular, the proposed design achieved average improvements in power consumption of up to 81%, 82.9%, and 40.6% for VGG-16, ResNet-18, and ResNet-50 workloads compared to the conventional bit-serial design, respectively. Furthermore, significant average speedups of 2.39×, 2.6×, and 2.42× were observed when comparing the proposed design to conventional bit-serial designs for the VGG-16, ResNet-18, and ResNet-50 models, respectively.

Optimizing Job Offloading Schedule for Collaborative DNN Inference

Deep Neural Networks (DNNs) have been widely deployed in mobile applications. DNN inference latency is a critical metric to measure the service quality of those applications. Collaborative inference is a promising approach for latency optimization, where partial inference workloads are offloaded from mobile devices to cloud servers. Model partition problems for collaborative inference have been well studied. However, little attention has been paid to optimizing offloading pipeline for multiple DNN inference jobs. In practice, mobile devices usually need to process multiple DNN inference jobs simultaneously. We propose to jointly optimize the DNN partitioning and pipeline scheduling for multiple inference jobs. We theoretically analyze the optimal scheduling conditions for homogeneous chain-structure DNNs. Based on the analysis, we proposed near-optimal partitioning and scheduling methods for chain-structure DNNs. We also extend those methods for general-structure DNNs. In addition, we extend our problem scenario to handle heterogeneous DNN inference jobs. A layer-level scheduling algorithm is proposed. Theoretical analyses show that our proposed method is optimal when computation graphs are tree-structure. Our joint optimization methods are evaluated in a real-world testbed. Experiment results show that our methods can significantly reduce the overall inference latency of multiple inference jobs compared to partition-only or schedule-only approaches.

StreamliNet: Cost-aware layer-wise neural network linearization for fast and accurate private inference

Private inference (PI) allows a client and a server to perform cryptographically-secure deep neural network inference without disclosing their sensitive data to each other. Despite the strong security guarantee, existing models are ill-suited for PI since their overused non-linear operations such as ReLUs are computationally expensive in the regime of ciphertext and therefore dominate the PI latency. Previous solutions on ReLU optimization either ignore the intrinsic importance of ReLU or suffer from significant accuracy loss. In this work, we propose StreamliNet, an importance-driven gradient-based framework to speed up PI latency and retain inference accuracy. Specifically, we first present a novel notion of ReLU negativity as a proxy for the ReLU importance in a multivariate metric to precisely identify layer-wise budgets. Then, our StreamliNet automates the selection of performance-insensitive ReLUs for linearization and learns the non-linearity sparse model where ReLUs are present in each layer with appropriate counts and locations. Moreover, in order to reduce the activation map discrepancy, we develop a cost-aware post-activation consistency constraint to prioritize the linearization of ReLUs with low cost while further mitigating the model performance degradation. Extensive experiments on various models and datasets demonstrate that StreamliNet outperforms the state-of-the-arts such as SNL (ICML 22) and SENet (ICLR 23) by boosting 3.09% more accuracy with iso-ReLU budget or requiring 2× fewer ReLUs with iso-accuracy, on CIFAR-100.

CUTE: A scalable CPU-centric and Ultra-utilized Tensor Engine for convolutions

Convolution is a fundamental and computationally expensive primitive and finds ubiquitous in deep neural networks (DNNs). The evolving DNNs have spurred the emergence of numerous accelerators and they successfully achieve high throughput. However, for DNN inference with small batch sizes, the computational resources of the accelerators are often under-utilized, and the overhead of offloading is significant. Compared to accelerators, the CPU can better meet fast response requirements of inference, flexibly handle various models, and is suitable for various scenarios (from edge to data center). Therefore, CPU remains an attractive platform for DNN inference, despite the sub-optimal performance, and resource efficiency.In this paper, we propose CUTE, a scalable CPU-centric and ultra-utilized tensor engine for convolutions. It co-designs data flow and hardware architecture to leverage the data reuse and parallelism of convolutions. CUTE is composed of several small tensor elements (TEs) and two-level buffers. It employs a decoupled access-execution architecture and greedy strategy to feed data to TEs, enabling it to achieve ultra utilization and great scalability. CUTE is tightly coupled with the CPU to minimize offloading latency, thereby providing efficient convolution computing capabilities for the system. Experimental results show that under the same bandwidth, CUTE achieves an average performance improvement of 3.8x compared with the CPU AVX512 unit and 1.6x compared with the CPU AMX unit. Besides, CUTE achieves a speedup of 7.0x and 3.9x over Nvidia V100 GPU and Eyeriss accelerator respectively, due to higher utilization of computing units.

Protecting Intellectual Property With Reliable Availability of Learning Models in AI-Based Cybersecurity Services

Artificial intelligence (AI)-based cybersecurity services offer significant promise in many scenarios, including malware detection, content supervision, and so on. Meanwhile, many commercial and government applications have raised the need for intellectual property protection of using deep neural network (DNN). Existing studies (e.g., watermarking techniques) on intellectual property protection only aim at inserting secret information into DNNs, allowing producers to detect whether the given DNN infringes on their own copyrights. However, since the availability protection of learning models is rarely considered, the piracy model can still work with high accuracy. In this paper, a novel model locking (M-LOCK) scheme for the DNN is proposed to enhance its availability protection, where the DNN produces poor accuracy if a specific token is absent, while it maps only the tokenized inputs into correct predictions. The proposed scheme performs the verification process during the DNN inference operation, actively protecting models' intellectual property copyright at each query. Specifically, to train the token-sensitive decision-making boundaries of DNNs, a data poisoning-based model manipulation (DPMM) method is also proposed, which minimizes the correlation between the dummy outputs and correct predictions. Extensive experiments demonstrate the proposed scheme could achieve high reliability and effectiveness across various benchmark datasets as well as typical model protection methods.

Functions as a service for distributed deep neural network inference over the cloud‐to‐things continuum

AbstractThe use of serverless computing has been gaining popularity in recent years as an alternative to traditional Cloud computing. We explore the usability and potential development benefits of three popular open‐source serverless platforms in the context of IoT: OpenFaaS, Fission, and OpenWhisk. To address this we discuss our experience developing a serverless and low‐latency Distributed Deep Neural Network (DDNN) application. Our findings indicate that these serverless platforms require significant resources to operate and are not ideal for constrained devices. In addition, we archived a 55% improvement compared to Kafka‐ML's performance under load, a framework without dynamic scaling support, demonstrating the potential of serverless computing for low‐latency applications.

Investigating the impact of transient hardware faults on deep learning neural network inference

SummarySafety‐critical applications, such as autonomous vehicles, healthcare, and space applications, have witnessed widespread deployment of deep neural networks (DNNs). Inherent algorithmic inaccuracies have consistently been a prevalent cause of misclassifications, even in modern DNNs. Simultaneously, with an ongoing effort to minimize the footprint of contemporary chip design, there is a continual rise in the likelihood of transient hardware faults in deployed DNN models. Consequently, researchers have wondered the extent to which these faults contribute to DNN misclassifications compared to algorithmic inaccuracies. This article delves into the impact of DNN misclassifications caused by transient hardware faults and intrinsic algorithmic inaccuracies in safety‐critical applications. Initially, we enhance a cutting‐edge fault injector, TensorFI, for TensorFlow applications to facilitate fault injections on modern DNN non‐sequential models in a scalable manner. Subsequently, we analyse the DNN‐inferred outcomes based on our defined safety‐critical metrics. Finally, we conduct extensive fault injection experiments and a comprehensive analysis to achieve the following objectives: (1) investigate the impact of different target class groupings on DNN failures and (2) pinpoint the most vulnerable bit locations within tensors, as well as DNN layers accountable for the majority of safety‐critical misclassifications. Our findings regarding different grouping formations reveal that failures induced by transient hardware faults can have a substantially greater impact (with a probability up to 4 higher) on safety‐critical applications compared to those resulting from algorithmic inaccuracies. Additionally, our investigation demonstrates that higher order bit positions in tensors, as well as initial and final layers of DNNs, necessitate prioritized protection compared to other regions.

A training method for deep neural network inference accelerators with high tolerance for their hardware imperfection

We propose a novel training method named hardware-conscious software training (HCST) for deep neural network inference accelerators to recover the accuracy degradation due to their hardware imperfections. Existing approaches to the issue, such as the on-chip training and the in situ training, utilize the forward inference data that are obtained by the inference accelerators for the backpropagation. In the approaches, since the memory devices that are used for the weights and the biases are to be switched after each epoch, the total number of the switching in the training process grows too large to avoid the problems of endurance limitation, nonlinearity and asymmetry in the switching of the nonvolatile memories used for the weights and the biases. The proposed training method is totally conducted by software whose forward inference path and backpropagation reflect the hardware imperfections, overcoming all the above problems. The HCST reformulates the mathematical expressions in the forward propagation and the gradient calculation with the backpropagation so that it replicates the hardware structure under the influence of variations in the chip fabrication process. The effectiveness of this approach is validated through the MNIST dataset experiments to manifest its capability to restore the accuracies. A circuit design is also disclosed for measuring the offset voltages and the open loop gains of the operational amplifiers used in the accelerator, showing that the chip area overhead is minor.

PArtNNer: Platform-Agnostic Adaptive Edge-Cloud DNN Partitioning for Minimizing End-to-End Latency

The last decade has seen the emergence of Deep Neural Networks (DNNs) as the de facto algorithm for various computer vision applications. In intelligent edge devices, sensor data streams acquired by the device are processed by a DNN application running on either the edge device itself or in the cloud. However, “edge-only” and “cloud-only” execution of State-of-the-Art DNNs may not meet an application’s latency requirements due to the limited compute, memory, and energy resources in edge devices, dynamically varying bandwidth of edge-cloud connectivity networks, and temporal variations in the computational load of cloud servers. This work investigates distributed (partitioned) inference across edge devices (mobile/end device) and cloud servers to minimize end-to-end DNN inference latency. We study the impact of temporally varying operating conditions and the underlying compute and communication architecture on the decision of whether to run the inference solely on the edge, entirely in the cloud, or by partitioning the DNN model execution among the two. Leveraging the insights gained from this study and the wide variation in the capabilities of various edge platforms that run DNN inference, we propose PArtNNer , a platform-agnostic adaptive DNN partitioning algorithm that finds the optimal partitioning point in DNNs to minimize inference latency. PArtNNer can adapt to dynamic variations in communication bandwidth and cloud server load without requiring pre-characterization of underlying platforms. Experimental results for six image classification and object detection DNNs on a set of five commercial off-the-shelf compute platforms and three communication standards indicate that PArtNNer results in 10.2× and 3.2× (on average) and up to 21.1× and 6.7× improvements in end-to-end inference latency compared to execution of the DNN entirely on the edge device or entirely on a cloud server, respectively. Compared to pre-characterization-based partitioning approaches, PArtNNer converges to the optimal partitioning point 17.6× faster.

TECO: A Unified Feature Map Compression Framework Based on Transform and Entropy.

The massive memory accesses of feature maps (FMs) in deep neural network (DNN) processors lead to huge power consumption, which becomes a major energy bottleneck of DNN accelerators. In this article, we propose a unified framework named Transform and Entropy-based COmpression (TECO) scheme to efficiently compress FMs with various attributes in DNN inference. We explore, for the first time, the intrinsic unimodal distribution characteristic that widely exists in the frequency domain of various FMs. In addition, a well-optimized hardware-friendly coding scheme is designed, which fully utilizes this remarkable data distribution characteristic to encode and compress the frequency spectrum of different FMs. Furthermore, the information entropy theory is leveraged to develop a novel loss function for improving the compression ratio and to make a fast comparison among different compressors. Extensive experiments are performed on multiple tasks and demonstrate that the proposed TECO achieves compression ratios of 2.31 × in ResNet-50 on image classification, 3.47 × in UNet on dark image enhancement, and 3.18 × in Yolo-v4 on object detection while keeping the accuracy of these models. Compared with the upper limit of the compression ratio for original FMs, the proposed framework achieves the compression ratio improvement of 21%, 157%, and 152% on the above models.

Satellite Video Remote Sensing for Flood Model Validation

AbstractSatellite‐based optical video sensors are poised as the next frontier in remote sensing. Satellite video offers the unique advantage of capturing the transient dynamics of floods with the potential to supply hitherto unavailable data for the assessment of hydraulic models. A prerequisite for the successful application of hydraulic models is their proper calibration and validation. In this investigation, we validate 2D flood model predictions using satellite video‐derived flood extents and velocities. Hydraulic simulations of a flood event with a 5‐year return period (discharge of 722 m3 s−1) were conducted using Hydrologic Engineering Center—River Analysis System 2D in the Darling River at Tilpa, Australia. To extract flood extents from satellite video of the studied flood event, we use a hybrid transformer‐encoder, convolutional neural network (CNN)‐decoder deep neural network. We evaluate the influence of test‐time augmentation (TTA)—the application of transformations on test satellite video image ensembles, during deep neural network inference. We employ Large Scale Particle Image Velocimetry (LSPIV) for non‐contact‐based river surface velocity estimation from sequential satellite video frames. When validating hydraulic model simulations using deep neural network segmented flood extents, critical success index peaked at 94% with an average relative improvement of 9.5% when TTA was implemented. We show that TTA offers significant value in deep neural network‐based image segmentation, compensating for aleatoric uncertainties. The correlations between model predictions and LSPIV velocities were reasonable and averaged 0.78. Overall, our investigation demonstrates the potential of optical space‐based video sensors for validating flood models and studying flood dynamics.

Scalable Heterogeneous Scheduling Based Model Parallelism for Real-Time Inference of Large-Scale Deep Neural Networks

Scalable Heterogeneous Scheduling Based Model Parallelism for Real-Time Inference of Large-Scale Deep Neural Networks

A Low-Latency DNN Accelerator Enabled by DFT-Based Convolution Execution Within Crossbar Arrays.

Analog resistive random access memory (RRAM) devices enable parallelized nonvolatile in-memory vector-matrix multiplications for neural networks eliminating the bottlenecks posed by von Neumann architecture. While using RRAMs improves the accelerator performance and enables their deployment at the edge, the high tuning time needed to update the RRAM conductance states adds significant burden and latency to real-time system training. In this article, we develop an in-memory discrete Fourier transform (DFT)-based convolution methodology to reduce system latency and input regeneration. By storing the static DFT/inverse DFT (IDFT) coefficients within the analog arrays, we keep digital computational operations using digital circuits to a minimum. By performing the convolution in reciprocal Fourier space, our approach minimizes connection weight updates, which significantly accelerates both neural network training and interference. Moreover, by minimizing RRAM conductance update frequency, we mitigate the endurance limitations of resistive nonvolatile memories. We show that by leveraging the symmetry and linearity of DFT/IDFTs, we can reduce the power by 1.57 × for convolution over conventional execution. The designed hardware-aware deep neural network (DNN) inference accelerator enhances the peak power efficiency by 28.02 × and area efficiency by 8.7 × over state-of-the-art accelerators. This article paves the way for ultrafast, low-power, compact hardware accelerators.

Hastening Stream Offloading of Inference via Multi-Exit DNNs in Mobile Edge Computing

As the primary driver of intelligent mobile applications, deep neural networks (DNNs) have gradually deployed to millions of mobile devices, producing massive latency-sensitive and computation-intensive tasks daily. Mobile edge computing facilitates the deployment of computing resources at the edge, which enables fine-grained offloading of DNN inference tasks from mobile devices to edge nodes. However, most existing studies have not systematically considered three crucial performance aspects: scheduling multiple streams of DNN inference tasks, leveraging multi-exit models to hasten task processing, and partitioning inference models for partial offloading. To this end, this paper proposes an adaptive inference framework in mobile edge computing, which can dynamically select the exit point and partition point for multiple inference task streams. We design a dynamic programming algorithm to obtain an efficient solution under the ideal condition that task arrival information is known. Further, we design a learning-based algorithm for online scheduling, whose training efficiency is improved based on historical experience initialization and priority experience replay. Experimental results show that compared with the Greedy algorithm, the online algorithm improves the performance on two environmental parameters by an average of 5.9% and 32%, respectively.

Advancements in Accelerating Deep Neural Network Inference on AIoT Devices: A Survey

Advancements in Accelerating Deep Neural Network Inference on AIoT Devices: A Survey

NeuroCool: Dynamic Thermal Management of 3D DRAM for Deep Neural Networks through Customized Prefetching

Deep neural network (DNN) implementations are typically characterized by huge datasets and concurrent computation, resulting in a demand for high memory bandwidth due to intensive data movement between processors and off-chip memory. Performing DNN inference on general-purpose cores/edge is gaining attraction to enhance user experience and reduce latency. The mismatch in the CPU and conventional DRAM speed leads to under-utilization of the compute capabilities, causing increased inference time. 3D DRAM is a promising solution to effectively fulfill the bandwidth requirement of high-throughput DNNs. However, due to high power density in stacked architectures, 3D DRAMs need dynamic thermal management (DTM), resulting in performance overhead due to memory-induced CPU throttling. We study the thermal impact of DNN applications running on a 3D DRAM system, and make a case for a memory temperature-aware customized prefetch mechanism to reduce DTM overheads and significantly improve performance. In our proposed NeuroCool DTM policy, we intelligently place either DRAM ranks or tiers in low power state, using the DNN layer characteristics and access rate. We establish the generalization of our approach through training and test datasets comprising diverse data points from widely used DNN applications. Experimental results on popular DNNs show that NeuroCool results in a average performance gain of 44% (as high as 52%) and memory energy improvement of 43% (as high as 69%) over general-purpose DTM policies.

Multiply-and-Fire: An Event-Driven Sparse Neural Network Accelerator

Deep neural network inference has become a vital workload for many systems from edge-based computing to data centers. To reduce the performance and power requirements for deep neural networks (DNNs) running on these systems, pruning is commonly used as a way to maintain most of the accuracy of the system while significantly reducing the workload requirements. Unfortunately, accelerators designed for unstructured pruning typically employ expensive methods to either determine non-zero activation-weight pairings or reorder computation. These methods require additional storage and memory accesses compared to the more regular data access patterns seen in structurally pruned models. However, even existing works that focus on the more regular access patterns seen in structured pruning continue to suffer from inefficient designs, which either ignore or expensively handle activation sparsity leading to low performance. To address these inefficiencies, we leverage structured pruning and propose the multiply-and-fire (MnF) technique, which aims to solve these problems in three ways: (a) the use of a novel event-driven dataflow that naturally exploits activation sparsity without complex, high-overhead logic; (b) an optimized dataflow takes an activation-centric approach, which aims to maximize the reuse of activation data in computation and ensures the data are only fetched once from off-chip global and on-chip local memory; and (c) based on the proposed event-driven dataflow, we develop an energy-efficient, high-performance sparsity-aware DNN accelerator. Our results show that our MnF accelerator achieves a significant improvement across a number of modern benchmarks and presents a new direction to enable highly efficient AI inference for both CNN and MLP workloads. Overall, this work achieves a geometric mean of 11.2× higher energy efficiency and 1.41× speedup compared to a state-of-the-art sparsity-aware accelerator.

Accelerating DNN Inference With Reliability Guarantee in Vehicular Edge Computing

This paper explores on accelerating Deep Neural Network (DNN) inference with reliability guarantee in Vehicular Edge Computing (VEC) by considering the synergistic impacts of vehicle mobility and Vehicle-to-Vehicle/Infrastructure (V2V/V2I) communications. First, we show the necessity of striking a balance between DNN inference acceleration and reliability in VEC, and give insights into the design rationale by analyzing the features of overlapped DNN partitioning and mobility-aware task offloading. Second, we formulate the Cooperative Partitioning and Offloading (CPO) problem by presenting a cooperative DNN partitioning and offloading scenario, followed by deriving an offloading reliability model and a DNN inference delay model. The CPO is proved as NP-hard. Third, we propose two approximation algorithms, i.e., Submodular Approximation Allocation Algorithm (SA <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^3$</tex-math> </inline-formula> ) and Feed Me the Rest algorithm (FMtR). In particular, SA <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$^3$</tex-math> </inline-formula> determines the edge allocation in a centralized way, which achieves 1/3-optimal approximation on maximizing the inference reliability. On this basis, FMtR partitions the DNN models and offloads the tasks to the allocated edge nodes in a distributed way, which achieves 1/2-optimal approximation on maximizing the inference reliability. Finally, we build the simulation model and give a comprehensive performance evaluation, which demonstrates the superiority of the proposed solutions.

Collaborative Inference Acceleration Integrating DNN Partitioning and Task Offloading in Mobile Edge Computing

In mobile edge computing environment, intelligent inference services driven by DNN are highly sensitive to latency. Recently, collaborative inference between User Devices and Edge Servers (ESs) based on Deep Neural Networks (DNN) partition has achieved success in service acceleration. However, most of the existing collaborative acceleration schemes are partitioned for a single DNN inference task, which cannot quickly make partition decisions for a set of concurrent inference tasks, and often sacrifice inference accuracy. In addition, due to the limited resources of ESs, there is resource competition among concurrent requests, which makes the partitioned tasks cannot be offloaded to ESs in time for processing. Therefore, designing an efficient offloading scheme becomes essential. The task offloading schemes based on deep reinforcement learning can solve complex decision-making problems in high-dimensional state space, but they have problems such as insufficient sample diversity and easily falling into local optimum. In this paper, a Collaborative Inference Acceleration Scheme integrating DNN Partitioning and Task Offloading (CIAS-PnO) is proposed. First, while ensuring inference accuracy, the Collaborative DNN Layer Partitioning (CDLP) algorithm is designed with the goal of optimal latency. CDLP can reduce the problem scale of concurrent inference tasks partition by pruning operation and determine the partition decisions in time. Then, the Distributed Soft Actor-Critic (SAC)-based Partition Task Offloading algorithm (DSACO) is designed. DSACO supports SAC Agents to explore samples in parallel and share learning experiences, and uses the automatic entropy adjustment mechanism to improve the exploration efficiency of Agents, so as to avoid falling into local optimum and achieve efficient offloading of partition tasks. Experimental results on DNN benchmarks show that compared with the baseline acceleration schemes, CIAS-PnO achieves more than 19.8% acceleration performance improvement, and has higher convergence performance and task success rate.